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Background / Context: 

At the federal and state levels, eeonomies has reeeived inereasing attention as a eritieal eontent 
area for K-12 edueation. In 1994 the Goals 2000 Edueate Ameriea Aet identified eeonomies as 
one of nine eore subjeet areas for developing eontent standards. Three years later, the National 
Connell on Eeonomie Edueation (NCEE) led a eoalition of organizations (ineluding the National 
Assoeiation of Eeonomie Edueators, the Eoundation for Teaehing Eeonomies, and the Ameriean 
Eeonomies Assoeiation’ s Committee on Eeonomie Edueation) to develop voluntary eontent 
standards to guide instruetion. The standards deseribe the eeonomies eontent for grades 1-12 and 
inelude 211 benehmarks detailing what students should know and be able to do (Siegfried and 
Meszaros 1998). Aeeording to the most reeentNCEE survey of 2007, 48 states now inelude 
eontent standards in eeonomies, with 40 requiring implementation of the standards, 23 requiring 
testing, and 17 requiring an eeonomies eourse for graduation (NCEE 2007). 

Even with the reeent national attention on eeonomies literaey in K-12 edueation (e.g. NAEP 
eeonomies test in 2006; EEE grant program in 2004 and 2005), there is less agreement about 
where eeonomies fits into the eurrieulum, effeetive ways of teaehing it, and how mueh subjeet- 
area baekground should be required of elassroom instruetors (Watts 2006). 

Watts (2006) reports that in states where eeonomies is required for high sehool graduation, it is 
typieally taught by following the state-adopted eontent standards, whieh are supported by a 
textbook. The format is generally one in whieh teaehers provide direet instruetion through a 
leeture format and eneourage student diseussion (see, for example, Mergendoller, Maxwell, and 
Bellisimo 2000). The teaehers’ objeetive is to follow the text from beginning to end, eovering 
eoneepts of theoretieal and applied miero- and maeroeeonomies. In praetiee, there is variation 
from elassroom to elassroom (Walstad 2001). Teaehers not only vary the sequeneing of the 
eourse, but also add eontent through lessons and aetivities to augment the textbook (Sehug, 
Dieterle, and Clark 2009). The variation is largely due to the faet that teaehers and their distriets 
remain ultimately responsible for designing the eurrieulum (Walstad 2001). 

In eontrast with the textbook-driven eurrieulum for high sehool eeonomies, the method tested in 
this experimental trial (Problem Based Eeonomies) uses a problem-based approaeh. Teaehers use 
eeonomie problems and follow a set of diseiplined and strategie analytie steps. The intent is that 
students learn to eontextualize, understand, reason, and solve what may at the outset have been a 
problem for whieh they had no analytie tools. 

Purpose / Objective / Research Question / Focus of Study: 

The primary purpose of this study is to assess student-level impaets of a problem-based 
instruetional approaeh to high sehool eeonomies. The eurrieulum approaeh examined here was 
designed to inerease elass partieipation and eontent knowledge for high sehool students who are 
learning eeonomies. This study tests the effeetiveness of Problem Based Eeonomies, developed 
by the Buek Institute for Edueation, on student learning of eeonomies eontent and problem- 
solving skills. Student aehievement outeomes are of primary importanee and are hypothesized to 
be mediated by ehanges in teaeher knowledge and pedagogieal praetiee. This study targeted high 
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schools in both urban and rural areas and engaged teaehers who eommitted to teach economics 
during the 2007/08 aeademie year. 

The researeh questions asked whether Problem Based Eeonomies changes: 

• Students’ content knowledge in eeonomies. 

• Students’ problem-solving skills in eeonomies. 

• Teaehers’ eontent knowledge in eeonomies. 

• Teaehers’ instruetional practices. 

• Teaehers’ satisfaction with teaching materials and methods used to teaeh economies. 

Setting: 

The field implementation for the study was high sehool elassrooms in partieipating 
sehools/distriets. The intervention group teaeher professional development took plaee in an 
office/eonferenee environment in the summer preeeding the 2007/2008 aeademie year. 

Population / Participants / Subjects: 

The study was implemented from summer 2007 through spring 2008 in high sehools in Arizona 
and California. For both of these states, high sehool eeonomies has beeome a required eourse for 
graduation and relevant to sehools and distriets as a result. Arizona targeted the graduating elass 
of 2009 as the first eohort of high sehool students that was required to eomplete a eourse in 
eeonomies; California has had this requirement in plaee sinee 2005. Study partieipants ineluded 
128 eeonomies teachers from 106 schools. Teachers were randomly assigned to the intervention 
or eontrol eondition (64 teaehers eaeh). The final analytie sample ineluded 4,350 students. Some 
39 pereent of the students were eligible for free or redueed-priee meals, 37 pereent were 
Hispanie, and 40 pereent were non-Hispanic White. 

Intervention / Program / Practice: 

The intervention for this study was a specifie set of Problem Based Eeonomies eurrieular 
materials provided to intervention group teaehers within a professional development and ongoing 
support program. The teaehers used the Problem Based Eeonomies materials as a major portion 
of their instruetional program in their high sehool eeonomies elasses in the 2007/08 aeademie 
year. 

A five-day professional development workshop provided to the intervention group familiarized 
teachers with the eurrieulum modules, using pedagogical strategies eonsistent with problem- 
based instruction. The Buek Institute provided trainers who were current or former eeonomies 
teaehers with substantial experienee using the Problem Based Eeonomies eurrieulum materials. 
The trainers reviewed one eurrieulum module eaeh day; pedagogieal strategies that are 
consistently applied in the units were modeled, highlighted, reinforeed, and discussed throughout 
the workshops. 

Following the professional development program, teaehers used the eurrieulum and associated 
methods to teaeh students in their elassrooms for a one-semester eourse. 
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The counterfactual for the study was typieal economies instruction using approved textbooks. 



Research Design: 

The study was designed as a within-school randomized controlled trial. 

The test of whether gains in economic literacy are seen between intervention and control students 
was accomplished by the administration of the Test of Economic Literacy (TEL), a 40-item 
closed-response economics exam (Walstad and Rebeck, 2001). The research team augmented 
this outcome measure with an opportunity to test students’ abilities to reason with the concepts 
they had learned. Each TEL item was rated “correcf’ (1 point) or “incorrect” (0 points); the 
possible overall TEL score ranged from 0 to 40. A set of “performance tasks”, developed by the 
University of California, Los Angeles’s National Center for Research on Education, Standards, 
and Student Testing (UCLA CRESST), gave students the ability to demonstrate problem-solving 
skills as they answered open-ended essay questions. The five assessment tasks used in this study 
focused on monetary policy/federal funds, monetary policy/employment, fiscal policy, consumer 
demand, and opportunity costs. Each student was randomly assigned two tasks. 

Both the TEL posttest and the performance task assessments were administered to the students 
by designated proctors (such as student counselors) at the end of the spring 2008 semester. See 
Table 1 for key study characteristics. 

Data Collection and Analysis: 

Data collection for student measures (pre-test) were collected by the economics teachers at the 
start of the semester. Posttest data for student measures were not collected by the teachers but 
rather by proctors identified in each school. Consistent with standardized test administration, 
proctors received instructions from the research team, including information on how to return 
test materials by secure mail for follow-up scoring. Proctors were either student counselors or 
school-level administrators familiar with proctoring examinations. Therefore, data collection was 
not blinded to assignment condition. Teacher-level data were collected by participating teachers, 
who received the instruments by mail along with preaddressed, stamped envelopes for returning 
them. 



Impacts of Problem Based Economics were estimated by comparing outcomes for students and 
teachers who were randomly assigned to the intervention and control groups. The impact 
analyses focused on the effect of the program on two primary student outcome domains 
(economics content knowledge and problem-solving skills) and three secondary teacher outcome 
domains (economics content knowledge, pedagogical practices, and satisfaction with teaching 
materials and methods). Eor student outcomes, the primary hypothesis-testing analyses involved 
fitting conditional multilevel regression models, with additional terms to account for the nesting 
of units within higher units of aggregation (Goldstein 1987; Raudenbush and Bryk 2002; Murray 
1998). A random effect for teachers was included in the model to account for the nesting of 
student observations within teachers. All outcome variables were treated as continuous variables 
in the impact analyses (estimated using multilevel or single-level linear regression models). To 
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increase the preeision of the estimates, a set of baseline characteristics of students and teachers 
was ineluded in the models as covariates. 

Findings / Results: 

The analysis at the primary (student) level supports the following: 

• A statistically significant finding that students whose teaehers had reeeived professional 
development and support in Problem Based Eeonomics (model-adjusted mean score = 22.61) 
outscored their control group peers (model-adjusted mean score = 20.01) on the TEL by an 
average of 2.6 test items (effect size = 0.32). 

• The outcomes on student measures of problem-solving skills and application to real-world 
economie dilemmas also showed signifieant differences in favor of the intervention group 
(model-adjusted mean score for the intervention group was 6.72 versus 6.18 for the eontrol 
group; the differenee of 0.54 corresponded to an effect size of 0.27). 

The study also confirmed the following at the secondary (teacher) level: 

• No statistieally significant difference between the intervention and eontrol groups on 
teaehers’ knowledge of economics (model-adjusted means were 37.15 and 36.86 for the 
intervention and control group teachers, respectively). As discussed in the conclusions of the 
report, a ceiling effect on the Test of Economic Literacy instrument may have masked any 
true content gains for teaehers. 

• No statistieally significant difference in teachers’ pedagogieal style with the survey measures 
used (model-adjusted means were 29.92 and 26.60 for the intervention and control group 
teachers, respectively). 

• Statistically significant differences in favor of the intervention group teaehers on a measure 
of satisfaction with the teaehing materials and methods (model-adjusted means were 8.35 and 
6.88 for the intervention and control group teachers, respectively; the difference of 1.47 
eorresponded to an effect size of 1.09). 

Conclusions: 

Educators may be looking for ways to strengthen their economics education programs. The 
findings of this study confirm that students benefited from the combination of the professional 
development program, ongoing support for teachers, and the Problem Based Economics 
curriculum. At the same time, teaehers reported satisfaction with the Problem Based Economics 
materials. Teaehers in the sample had, on average, fewer than three college-level economies 
courses; the opportunity to engage in a five-day workshop in economies instruction offers 
support to teachers who are interested in advaneing their own professional development and 
increasing content knowledge.. 

During recruitment, the majority of teaehers who agreed to partieipate in the study expressed 
enthusiasm for the material they taught, its relevance to students’ lives, and the idea that a 
researeh study would benefit the profession broadly. Recruitment for the study was not easy, 
however. Hundreds of economics teachers deelined to participate. The original 128 who agreed 
to participate were interested in finding better ways to reaeh their students. They included both 
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new and seasoned teaehers, with some variation in eontent expertise. What they had in eommon 
was their willingness to partieipate in the experiment — a selection bias that could not be 
quantified, but must be acknowledged. This has implications for the generalizability of the study, 
The results of this study are likely to apply mainly to teachers and schools where the economics 
program and the associated professional development are a priority. From the perspective of the 
students, we also note that their participation in the study was voluntary; we cannot quantify 
whether students unwilling to participate in the economics tests would have performed 
differently than the study sample described in this report. 
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Appendix B, Tables and Figures 

Not included in page count. 



Table 1. Study characteristics and data collection schedule for high school instruction with Problem 
Based Economics 



Study design 


Cluster-randomized trial 


Unit of assignment 


Teachers 


Statistical power estimates 


For Type 1 error = .05, 80 percent or higher power to detect 
minimum detectable effect size of 0.18-0.21 at student level 
and 0.55 at teacher leveE 


Implementation began 


Summer 2007 


Student measures 




Test of Economic Literacy (pre/post) 
Student surveys (pre/post) 
Performance task assessments 


Administered January 2008, June 2008 
Administered January 2008, June 2008 
Administered June 2008 


Teacher measures 




Test of Economic Literacy (pre/post) 
Teacher surveys (pre/post) 


Administered June-August 2007, June 2008 
Administered June-August 2007, June 2008 



Note.' a. The estimates were based on 83 teachers, with an average of 40 students per teacher. The study team closely worked 
with these teachers to collect data throughout the study period. The detailed flow of the teacher sample is presented later in this 
chapter (figure 2.1). The intraclass correlation was assumed to be either 0.15 or 0.20. Appendix A provides the power estimates 
based on the final analytic samples. 
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